Method for managing annotation job, apparatus and system supporting the same

ABSTRACT

A computing device obtains information about a medical slide image, and determines a dataset type of the medical slide image and a panel of the medical slide image. The computing device assigns to an annotator account, an annotation job defined by at least the medical slide image, the determined dataset type, an annotation task, and a patch that is a partial area of the medical slide image. The annotation task includes the determined panel, and the panel is designated as one of a plurality of panels including a cell panel, a tissue panel, and a structure panel. The dataset type indicates a use of the medical slide image and is designated as one of a plurality of uses including a training use of a medical learning model and a validation use of the machine learning model.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation Application of U.S. patentapplication No. 16/671,430, which was filed on Nov. 1, 2019, which is acontinuation of PCT Application No. PCT/KR2018/013664 filed on Nov. 9,2018, the entire contents of which are incorporated herein by reference.

BACKGROUND (a) Field

The present disclosure relates to an annotation job management method,an apparatus and a system supporting the same. In particular, thepresent disclosure relates to a method for managing annotation job moreefficiently and ensuring accuracy of annotation results, and anapparatus and a system for supporting the method.

(b) Description of the Related Art

As shown in FIG. 1 , supervised learning is a machine learning methodfor constructing a target model 3 for performing a target task bylearning a dataset 2 having label information (that is, correct answerinformation). Therefore, in order to perform the supervised learning ona dataset 1 which doesn't have label information (indicated by a tagicon), annotation job should precede.

The annotation job means a job of tagging label information for eachdata to generate a training dataset. Since the annotation job wasgenerally performed manually, a considerable amount of human cost andtime cost is consumed for generating a large amount of training dataset.In particular, in case of building a machine learning model fordetecting the location of lesions or determining the lesion types inmedical images, the annotation job should be performed by a skilledpractitioner, which is much more expensive than other domains.

In the past, the annotation job was not performed according to asystematic process. For example, in the conventional method, todetermine whether to perform an annotation, an administrator identifiesthe characteristics of medical images with eyes, classifies the medicalimages manually, and then assigns each medical image to an appropriateannotator. In addition, in the related art, the administrator designatesa region for annotation on a medical image, and then assigns a task toan annotator. That is, various processes such as medical imageclassification, job assignment, annotation region designation, and thelike are manually performed by an administrator, which causes aconsiderable amount of time cost and human cost to be consumed for theannotation job.

Furthermore, although the machine learning technique itself has beensufficiently advanced, there have been many difficulties in applyingmachine learning techniques to various fields due to the time and costproblems of the annotation job.

Therefore, in order to increase the usability of machine learningtechniques, a method for performing annotation more efficiently andsystematically is required.

SUMMARY

Some embodiment of the present disclosure provide a method for moreefficiently and systematically performing and managing annotation jobthrough annotation job automation, and an apparatus and systemsupporting the method.

Some embodiments of the present disclosure provide a data design productor a data modeling product that can systematically manage annotationjobs.

Some embodiments of the present disclosure provide a method forautomatically assigning an annotation job to an appropriate annotator,and an apparatus and system supporting the method.

Some embodiments of the present disclosure provide a method forautomatically extracting a patch image where an annotation job isperformed from a medical slide image, and an apparatus and systemsupporting the method.

Some embodiments of the present disclosure provide a method for ensuringthe accuracy of annotation results, and an apparatus and systemsupporting the method thereof.

It should be noted that objects of the present disclosure are notlimited to the above-described object, and other objects of the presentdisclosure will be apparent to the person of ordinary skill in the artfrom the following descriptions.

According to an embodiment of the present invention, an annotation jobmanagement method performed by a computing device may be provided. Themethod may include obtaining information about a medical slide image,determining a dataset type of the medical slide image and a panel of themedical slide image, and assigning to an annotator account, anannotation job defined by at least the medical slide image, thedetermined dataset type, an annotation task and a patch that is apartial area of the medical slide image. The annotation task may includethe determined panel, and the panel may be designated as one of aplurality of panels including a cell panel, a tissue panel, and astructure panel. The dataset type may indicate a use of the medicalslide image and may be designated as one of a plurality of usesincluding a training use of a machine learning model or a validation useof the machine learning model.

In some embodiments, the annotation task may be defined to furtherinclude a task class. The task class may indicate an annotation targetdefined from a perspective of the panel.

In some embodiments, the plurality of uses may further include an OPT(Observer Performance Test) use of the machine learning model.

In some embodiments, determining the dataset type and panel may includeinputting the medical slide image to the machine learning model anddetermining the dataset type and the panel based on an output value ofthe machine learning model.

In some embodiments, obtaining the information about the medical slideimage may include detecting, by a worker agent monitoring a storage,that a file of the medical slide image is added to a designated locationon the storage, inserting, by the worker agent, information about themedical slide image into a database, and obtaining the information aboutthe medical slide image from the database.

In some embodiments, assigning to the annotator account the annotationjob may include automatically assigning the annotation job to theannotator account that who is selected based on an annotationperformance history associated with a combination of the dataset type ofthe annotation job and the panel of the annotation task.

In some embodiments, the annotation task may further include a taskclass. The task class may indicate an annotation target defined from aperspective of the panel. Further, assigning to the annotator accountthe annotation job may include automatically assigning the annotationjob to the annotator account that is selected based on an annotationperformance history associated with a combination of the panel and thetask class of the annotation task.

In some embodiments, assigning to the annotator account the annotationjob may include obtaining candidate patches of the medical slide image,inputting each of the candidate patches to the machine learning model toobtain an output value for each class, and automatically selecting apatch for the annotation job among the candidate patches based on theoutput value for each class.

In some embodiments, automatically selecting the patch for theannotation job may include calculating an entropy value using the outputvalue for each class for each of the candidate patches and selecting apatch for the annotation job having the entropy value being equal to orgreater than a reference value from among the candidate patches.

In some embodiments, assigning to the annotator account the annotationjob may include obtaining candidate patches of the medical slide image,calculating a misprediction probability of the machine learning modelfor each of the candidate patches, and selecting as a patch of theannotation job a candidate patch having the calculated mispredictionprobability being equal to or greater than a reference value from amongthe candidate patches.

In some embodiments, the annotation job management method may furtherinclude obtaining a first annotation result data of the annotatoraccount assigned the annotation job, comparing the first annotationresult data with a result that is obtained by inputting a patch of theannotation job to the machine learning model, and reassigning theannotation job to another annotator account when a difference betweenthe two results is greater than a reference value.

In some embodiments, the annotation job management method may furtherinclude obtaining a first annotation result data of the annotatoraccount assigned the annotation job, obtaining a second annotationresult data of another annotator account, and disapproving the firstannotation result data when a similarity between the first annotationresult data and the second annotation result data is less than areference value.

According to another embodiment, an annotation management apparatusincluding a memory including one or more instructions and a processor isprovided. The processor, by executing the one or more instructions, mayobtain information about a medical slide image, determines a datasettype of the medical slide image and a panel of the medical slide image,and assigns to an annotator account, an annotation job which is definedby at least the medical slide image, the determined dataset type, anannotation task and a patch that is a partial area of the medical slideimage. The annotation task may include the determined panel, and thepanel may be designated as one of a plurality of panels including a cellpanel, a tissue panel and a structure panel. The dataset type mayindicate a use of the medical slide image and may be designated as oneof a plurality of uses including a training use of a machine learningmodel and a validation use of the machine learning model.

In some embodiments, a non-transitory computer-readable medium storing acomputer program is provided. The computer program, when executed by acomputing device, may cause the computing device to obtain informationabout a medical slide image, determine a dataset type of the medicalslide image and a panel of the medical slide image, and assign to anannotator account, an annotation job defined by at least the medicalslide image, the determined dataset type, an annotation task and a patchthat is a partial area of a medical slide image. The annotation task mayinclude the determined panel, and the panel may be designated as one ofa plurality of panels including a cell panel, a tissue panel, and astructure pane, The dataset type may indicate a use of the medical slideimage and may be designated as one of a plurality of uses including atraining use of a medical learning model and the validation use of amedical learning model.

According to various embodiments of the present disclosure as describedabove, as the annotation task is automated generally, the convenience ofthe administrator may be increased, and the overall work efficiency maybe greatly improved. Accordingly, the time cost and human cost requiredfor the annotation job may be greatly reduced. In addition, as theburden of annotation job is reduced, the utility of machine learningtechniques may be further increased.

In some embodiment, various data associated with the annotation job maybe systematically managed based on the data modeling output. As aresult, data management costs can be reduced, and the overall annotationjob process may be facilitated.

In some embodiment, by automatically assigning annotation jobs toappropriate annotators, the burden on the administrator may be reduced,and the accuracy of annotation results can be improved.

In some embodiment, the accuracy of the annotation results may beensured by comparing and validating the results of annotation jobs withthose of machine learning models or of other annotators. Accordingly,the performance of the machine learning model that learns the annotationresults may also be improved.

In some embodiment, a patch that indicates an area where annotation isperformed may be automatically extracted. Therefore, the workload of theadministrator may be minimized

In some embodiment, only patches effective for learning, among aplurality of candidate patches, may be selected as annotation targetsbased on misprediction probabilities, entropy values, and the like ofthe machine learning model. As a result, the amount of annotation jobsmay be reduced, and a good learning dataset may be generated.

The effects of the present disclosure are not limited to those mentionedabove, and other effects which have not been mentioned can be clearlyunderstood by the person of ordinary skill in the art from the followingdescription.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example diagram for explaining a relationship betweensupervised learning and an annotation job.

FIG. 2 and FIG. 3 are example diagrams for explaining an annotation jobmanagement system according to some embodiments of the presentdisclosure.

FIG. 4 is a diagram of an example data model for annotation jobmanagement according to some embodiments of the present disclosure.

FIG. 5 is an example flowchart showing an annotation job managementmethod according to some embodiments of the present disclosure.

FIG. 6 is an example diagram for explaining an annotator selectionmethod according to some embodiments of the present disclosure.

FIG. 7 is an example diagram showing an annotation tool to be referredin some embodiments of the present disclosure.

FIG. 8 is an example flowchart showing an annotation job generationmethod according to some embodiments of the present disclosure.

FIG. 9 is an example flowchart showing a dataset type determinationmethod for a medical slide image according to some embodiments of thepresent disclosure.

FIG. 10 , FIG. 11 , FIG. 12 , and FIG. 13 are example diagrams forexplaining a panel type determination method according to someembodiments of the present disclosure.

FIG. 14 is an example flowchart showing an automatic patch extractionmethod according to a first embodiment of the present disclosure.

FIG. 15 , FIG. 16 , FIG. 17 , FIG. 18 , and FIG. 19 are example diagramsfor explaining an automatic patch extraction method according to a firstembodiment of the present disclosure.

FIG. 20 is an example flowchart showing an automatic patch extractionmethod according to a second embodiment of the present disclosure.

FIG. 21 , FIG. 22 , and FIG. 23 are example diagrams for explaining anautomatic patch extraction method according to a second embodiment ofthe present disclosure.

FIG. 24 is an example computing device for implanting anapparatus/system according to various embodiments of the presentdisclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, preferred embodiments of the present disclosure will bedescribed with reference to the attached drawings. Advantages andfeatures of the present disclosure and methods of accomplishing the samemay be understood more readily by reference to the following detaileddescription of preferred embodiments and the accompanying drawings. Thepresent disclosure may, however, be embodied in many different forms andshould not be construed as being limited to the embodiments set forthherein. Rather, these embodiments are provided so that this disclosurewill be thorough and complete and will fully convey the concept of thedisclosure to the person of ordinary skill in the art, and the presentdisclosure will only be defined by the appended claims. Like referencenumerals designate like elements throughout the specification.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by theperson of ordinary skill in the art to which this disclosure belongs.Further, it will be further understood that terms, such as those definedin commonly used dictionaries, should be interpreted as having a meaningthat is consistent with their meaning in the context of the relevant artand the present disclosure, and will not be interpreted in an idealizedor overly formal sense unless expressly so defined herein. The termsused herein are for the purpose of describing particular embodimentsonly and is not intended to be limiting. As used herein, the singularforms “a”, “an” and “the” are intended to include the plural forms aswell, unless the context clearly indicates otherwise.

It will be further understood that, although the terms first, second, A,B, (a), (b), and the like may be used herein to describe variouselements, components, steps and/or operations. These terms are only usedto distinguish one element, component, step or operation from anotherelement, component, step, or operation. Thus, a first element,component, step or operation discussed below could be termed a secondelement, component, step or operation without departing from theteachings of the present inventive concept. It will be furtherunderstood that when an element is referred to as being “connected to”or “coupled with” another element, it can be directly connected orcoupled with the other element or intervening elements may be present.

It will be further understood that the terms “comprise” or “comprising”,“include” or “including”, and “have” or “having” specify the presence ofstated elements, steps, operations, and/or devices, but do not precludethe presence or addition of one or more other elements, steps,operations, and/or devices.

Before description of this specification, some terms used herein will beclarified.

As used herein, the term “label information” is correct answerinformation of a data sample and refers to information obtained throughan annotation job. The term “label” may be used interchangeably withterms such as annotation and tag.

As used herein, the term “annotation” refers to a job for tagging labelinformation on a data sample or tagged information (i.e., annotation)itself. The term “annotation” may be used interchangeably with termssuch as tagging and labeling.

As used herein, the term “misprediction probability” refers to aprobability or a possibility that a prediction result includes an error(that is, a probability that a prediction is incorrect), when a specificmodel for a given data sample performs prediction.

As used herein, the term “panel” refers to a type of patch to beextracted from a medical slide image or a type of the medical slideimage itself. The panel may be classified into a cell panel, a tissuepanel, and a structure panel, but the technical scope of the presentdisclosure is not limited thereto. For reference, examples of patchescorresponding to each type of panels are shown in FIG. 10 to FIG. 12 .

As used herein, the term “instruction” refers to a series ofinstructions that are grouped by function and are executed by aprocessor or a component of a computer program.

Hereinafter, some embodiments of the present disclosure are described indetail with reference to the accompanying drawings.

FIG. 2 is an example diagram for explaining an annotation job managementsystem according to some embodiments of the present disclosure.

Referring to FIG. 2 , an annotation job management system may include astorage server 10, one or more annotator terminals 20-1 to 20-n, and anannotation job management apparatus 100. However, the system shown inFIG. 1 merely corresponds to an exemplary embodiment for achieving anobject of the present disclosure, and some elements may be added oromitted as necessary. For example, in some other embodiments, as shownin FIG. 3 , the annotation job management system may further include areviewer terminal 30 where review (that is, evaluation) on theannotation job is performed.

The elements of the system shown in FIG. 2 or FIG. 3 representfunctional elements that are classified functionally. A plurality ofelements may be implemented as an integrated form in the physicalenvironment. Alternatively, each element may be segmented into aplurality of elements according to detailed functions in the physicalenvironment. For example, a first function of the annotation jobmanagement apparatus 100 may be implemented in a first computing deviceand a second function of the annotation job management apparatus 100 maybe implemented in a second computing device. Hereinafter, each elementis described in more detail with reference to FIG. 2 and FIG. 3 .

In the annotation job management system, the storage server 10 is aserver that stores and manages various data associated with anannotation job. In order to manage data effectively, the storage server10 may store and manage the various data using a database.

The various data may include a medical slide image file, metadata of themedical slide image (e.g. an image format, a name of related disease, arelated tissue, related patient information, and the like), data aboutthe annotation job, data about an annotator, and a result of theannotation job. However, the technical concept of the present disclosureis not limited thereto.

In some embodiments, the storage server 10 may function as a web serverthat presents a job management web page. In this case, through the jobmanagement web page, an administrator may perform job management such asassigning an annotation job, and the annotator may recognize and performthe assigned job.

In some embodiments, a data model (e.g. DB schema) for the annotationjob management may be designed as shown in FIG. 4 . In FIG. 4 , abox-shaped object refers to an entity, a line connecting box-shapedobjects refers to a relationship, and letters on the line refer to arelationship type. As shown in FIG. 4 , an annotation job entity 44 maybe associated with various entities 43, 45, 46, 47, and 49. Hereinafter,for convenience of understanding, the data model shown in FIG. 4 isbriefly described centered on the job entity 44.

A slide entity 45 is an entity related to a medical slide image. Theslide entity 45 may have various information associated with the medicalslide image as attributes. Since a plurality of annotation jobs may begenerated from one medical slide image, a relationship between the slideentity 45 and the job entity 44 is 1:n.

A dataset entity 49 is an entity that represents a use of the annotateddata. For example, the use may be classified into a training use (thatis, utilized as a training dataset), a validation use (that is, utilizedas a validation dataset), a test use (that is, utilized as a testdataset), or an OPT (Observer Performance Test) use (that is, utilizedin the OPT test), but the technical concept of the present disclosure isnot limited thereto.

An annotator entity 47 is an entity that represents an annotator. Theannotator entity 47 may have as attributes, such as a current job statusof the annotator, a job history performed previously, an evaluationresult of the previously-performed job, personal information of theannotator (e.g., education, major, etc.), and the like. Since oneannotator can perform multiple jobs, a relationship between theannotator entity 47 and the job entity 44 is 1:n.

A patch entity 46 is an entity associated with a patch derived from themedical slide image. Since the patch may include a plurality ofannotations, a relationship between the patch entity 46 and anannotation entity 48 is 1:n. Furthermore, since one annotation job maybe performed on a plurality of patches, a relationship between the patchentity 46 and the job entity 44 is n:1.

An annotation task entity 43 represents an annotation task that is adetailed type of the annotation job. For example, the annotation task isdefined and categorized as a task of tagging whether a specific cell isa mitotic cell, a task of tagging the number of mitotic cells, a task oftagging a type of a lesion, a task of tagging a location of the lesion,a task of tagging a name of disease, or the like. The detailed types ofthe annotation job can vary according to panels (that is, an annotationtagged on the cell panel may be different from an annotation tagged onthe tissue panel). Since different tasks can be performed on the samepanel, the task entity 43 may have a panel entity 41 and a task classentity 42 as attributes. Here, the task class entity 42 represents anannotation target (e.g., the mitotic cell, the location of the lesion)defined from a perspective of the panel or an annotation type definedfrom a perspective of the panel. Since multiple annotation jobs can begenerated from one annotation task (that is, there can be multiple jobsperforming the same task), a relationship between the annotation taskentity 43 and the annotation job entity 44 is 1:n. From a programmingpoint of view, the annotation task entity 43 may correspond to a classor a program, and the annotation job entity 44 may correspond to aninstance of the class or to a process generated through execution of theprogram.

In some embodiments, the storage server 10 may build a database based onthe data model described above, and may systematically manage variousdata associated with the annotation job. As a result, the datamanagement cost can be reduced and the overall work process ofannotation can be facilitated.

The data model for the annotation job management has been describedabove. Next, each element of the annotation job management system iscontinuously described with reference to FIG. 2 and FIG. 3 .

In the annotation job management system, the annotation job managementapparatus 100 is a computing device that performs various managementfunctions such as assigning annotation jobs to the annotator terminals20-1 to 20-n. Here, the computing device may be a tablet, a desktop, alaptop, or the like but is not limited thereto, and may include any kindof device having computing functions. An example of the computing devicewill be described with reference to FIG. 24 . In the followingdescription, for convenience of description, the annotation jobmanagement apparatus 100 is referred to as a “management apparatus” 100.In addition, when annotator terminals are designated collectively or acertain annotator terminal is designated, a reference numeral 20 isused.

The job management apparatus 100 may be a device used by anadministrator. For example, through the job management apparatus 100,the administrator may access the job management web page, log in with anadministrator account, and then perform overall management on theannotation jobs. For example, the manager may perform management such asassigning an annotation job to a specific annotator or requesting areview by transmitting the annotation result to the reviewer's account.The overall management process as described above may be automaticallyperformed by the job management apparatus 100, which will be describedwith reference to FIG. 5 .

In the annotation job management system, the annotator terminal 20 is aterminal on which the annotation task is performed by the annotator. Anannotation tool may be installed in the terminal 20. Various functionsfor annotation may be provided through the job management web page. Inthis case, the annotator may access the job management web page throughthe terminal 20 and then perform the annotation job on the web. Anexample of the annotation tool is shown in FIG. 7 .

In the annotation job management system, the reviewer terminal 30 is aterminal of the reviewer to perform a review on the annotation result.The reviewer may review the annotation result using the reviewerterminal 30, and provide a review result to the management apparatus100.

In some embodiments, at least some elements of the annotation jobmanagement system may communicate over a network. Here, the network maybe implemented by using any type of wired/wireless network such as alocal area network (LAN), a wide area network (WAN), a mobile radiocommunication network, a wireless broadband Internet (Wibro), or thelike.

The annotation job management system according to some embodiments ofthe present disclosure has been described above with reference to FIG. 2to FIG. 4 . Hereinafter, an annotation job management method accordingto some embodiments of the present disclosure is described withreference to FIG. 5 to FIG. 23 .

Each step of the annotation job management method may be performed by acomputing device. In other words, each step of the annotation jobmanagement method may be implemented as one or more instructions to beexecuted by a processor of the computing device. For convenience ofunderstanding, the description is continued on the assumption that theannotation job management method is performed in an environment shown inFIG. 3 or FIG. 4 .

FIG. 5 is an example flowchart showing an annotation job managementmethod according to some embodiments of the present disclosure. Theflowchart merely corresponds to an example embodiment for achieving anobject of the present disclosure, and some steps may be added or omittedas necessary.

As shown in FIG. 5 , the annotation job management method begins withstep S100 of acquiring information about a new medical slide image. Theinformation about the medical slide image may include metadata of themedical slide image, and may further include a medical slide image file.

In some embodiments, the information about the new medical slide imagemay be obtained through a worker agent in real time. More specifically,the worker agent may detect that the medical slide image file is addedon the storage that is designated by the worker agent (that is, astorage server 10 or a storage of a medical institution providing themedical slide image). In addition, the worker agent may insert theinformation about the new medical slide image into a database of the jobmanagement apparatus 100 or the storage server 10. Then, the informationabout the new medical slide image may be obtained from the database.

In step S200, the management apparatus 100 generates an annotation jobfor the medical slide image. Here, the annotation job may be definedbased on information such as the medical slide image, a dataset type, anannotation task, and a patch which is a partial area of the medicalslide image (that is, an annotation target area) (refer to FIG. 4 ). Adetailed description of step S200 will be described with reference toFIG. 8 to FIG. 23 .

In step S300, the management apparatus 100 selects an annotator toperform the generated annotation job.

In some embodiments, as shown in FIG. 6 , the management apparatus 100may select the annotator based on management information such as a jobperformance history 54 (e.g., annotation jobs frequently performed by aspecific annotator), evaluation results (or verification results) 55 ofpreviously-performed jobs, and a current job status 56 (e.g., a progressstatus of the currently assigned job). For example, the managementapparatus 100 may select a first annotator who has frequently performeda job associated with the generated annotation job, a second annotatorwith high performance on the job associated with the generatedannotation job, or a third annotator who currently has a small workload,as the annotator for the generated annotation job.

In one embodiment, whether the job performance history includes the jobassociated with the generated annotation job may be determined based onwhether a combination of a dataset type and a panel of an annotationtask in each job is similar to a combination of those in the generatedannotation job. In another embodiment, it may be determined based onwhether a combination of the panel and a task class of the annotationtask in each job is similar to a combination of those in the generatedannotation job. In yet another embodiment, it may be determined based onwhether the above-described two combinations are similar to those in thegenerated annotation job.

In some embodiments, when the new medical slide image is significantdata (e.g. a slide image associated with rare disease, a high qualityslide image, and the like.), a plurality of annotators may be selected.In addition, the number of the annotators may be increased in proportionto the significance. In this case, verification of the annotationresults may be performed by comparing the respective annotation resultsof the plurality of annotators with each other. According to the presentembodiment, through more strict verification on the significant data,the accuracy of annotation results can be improved.

In step S400, the management apparatus 100 assigns the annotation job tothe terminal 20 of the selected annotator. For example, the managementapparatus 100 may assign the annotation job to an account of theselected annotator.

In step S500, an annotation is performed on the annotator terminal 20.The annotator may perform the annotation using an annotator toolprovided in the terminal 20 or an annotation service provided throughweb (e.g., a job management web page), but the technical scope of thepresent disclosure is not limited thereto.

Some examples of the annotator tools are shown in FIG. 7 . As shown inFIG. 7 , the annotation tool 60 may include a first area 63 and a secondarea 61. The second area 61 may include a patch area 68 on which anannotation is performed and an enlargement/reduction indicator 65. Asshown in FIG. 7 , the patch area 68 may be highlighted, for example, byusing a box line or the like. Job information 67 is displayed in thefirst area 63, and a tool area 69 may be further included in the firstarea 63. The tool area 69 may include selectable tools corresponding toeach type of annotation. Thus, each annotator can tag an annotation inthe patch area 68 simply using a selected tool (e.g., perform tagging byselecting a first tool by click and then clicking the patch area 68)without directly noting the annotation in the patch area 68. Since thetype of annotation displayed in the tool area 63 may vary depending onthe annotation job, the annotation tool 60 may set appropriateannotation tools based on the information about the annotation job.

It should be understood that the annotation tool 60 shown in FIG. 7 onlyshows one example of a tool designed for convenience of the annotator.In other words, the annotation tool can be implemented in any way. Thedescription is continued with reference to FIG. 5 again.

In step S600, the annotator terminal 20 provides a result of theannotation job. The result of the annotation job may be labelinformation tagged to the corresponding patch.

In step S700, the management apparatus 100 performs validation(evaluation) on the job result. The validation result may be recorded asan evaluation result of the annotator. A method for validation may varyaccording to embodiments.

In some embodiments, the validation may be performed based on an outputof a machine learning model. Specifically, when a first annotationresult data is acquired from the annotator assigned to the job, thefirst annotation result data may be compared with a result obtained byinputting a patch of the annotation job to the machine learning model.As a result of the comparison, if it is determined that a differencebetween the two results exceeds a reference value, the first annotationresult data may be suspended or disapproved.

In one embodiment, the reference value may be a predetermined fixedvalue or a variable value according to a condition. For example, thereference value may be smaller as the accuracy of the machine learningmodel is higher.

In step S800, the management apparatus 100 determines whether theannotation job should be performed again. For example, when thevalidation result indicates that the annotation job is not performedsuccessfully in step S700, the management apparatus 100 may determinethat the annotation job needs to be performed again.

In step S900, in response to the determination, the management apparatus100 selects other annotator and reassigns the annotating job to theother annotator. In this case, the other annotator may be selectedthrough a similar way as described in step S300. Alternatively, theother annotator may be a reviewer or a machine learning model having thebest performance.

Although not shown in FIG. 5 , after step S900, the validation of thefirst annotation result data may be performed again based on a secondannotation result data of the other annotator. In detail, when thesecond annotation result data is acquired, a similarity between thefirst annotation result data and the second annotation result data maybe calculated. In addition, when the similarity is less than a referencevalue, the first annotation result data may be finally disapproved. Suchthe result may be recorded in the job performance history of theannotator.

An annotation job management method according to some embodiments of thepresent disclosure has been described above with reference to FIG. 4 toFIG. 7 . According to the above-described embodiments, as the wholeannotation job process can be automated, so that the convenience of theadministrator can be increased and the overall work efficiency can begreatly improved. Accordingly, the time cost and human cost required forthe annotation job can be greatly reduced. In addition, as the burden ofannotation job is reduced, the usability of the machine learningtechnique can be further increased.

Furthermore, the accuracy of the annotation results can be ensured bycomparing and validating the results of the annotation job with those ofthe machine learning model or those of other annotators. Accordingly,the performance of the machine learning model that learns the annotationresults can also be improved.

Hereinafter, a detailed process of the annotation job generation stepS200 is described with reference to FIG. 8 to FIG. 22 .

FIG. 8 is an example flowchart showing a method of generating anannotation job according to some embodiments of the present disclosure.However, the flowchart merely corresponds to an exemplary embodiment forachieving the object of the present disclosure, and some steps may beadded or omitted as necessary.

As shown in FIG. 8 , the method of generating an annotation job beginswith step 5210 of determining a dataset type of a new medical slideimage. As described above, the dataset type indicates a use of themedical slide image, and the use may be classified into a training use,a validation use, a test use, or observer performance test (OPT) use.

In some embodiments, the dataset type may be determined by theadministrator.

In some embodiments, the dataset type may be determined based on aconfidence score of a machine learning model for medical slide images.Here, the machine learning model refers to a model (that is, a learningtarget model) that performs a specific task (that is. lesionclassification, lesion location recognition, or the like) based on themedical slide images. Details of the present embodiment are shown inFIG. 9 . As shown in FIG. 9 , a management apparatus 100 inputs themedical slide image into the machine learning model, acquires theconfidence score as a result (S211), and determines whether theconfidence score is more than or equal to a reference value (S213). Inresponse to the determination that confidence score is less than thereference value, the management apparatus 100 determines that thedataset type of the medical slide image is the training use (S217). Theconfidence score being less than the reference value means that themachine learning model does not clearly interpret the medical slideimage (that is, training for the medical slide image is necessary).Otherwise, the dataset type of the medical slide image is determined tobe the validation use (or test use) (S215).

In some embodiments, the dataset type may be determined based on anentropy value of the machine learning model for the medical slide image.The entropy value is an indicator of uncertainty and may have a largervalue as the confidence scores are distributed more evenly over theclasses. In this embodiment, in response to the determination that theentropy value is more than or equal to the reference value, the datasettype may be determined to the training use. Otherwise, it may bedetermined to the validation use.

Referring back to FIG. 8 , in step S230, the management apparatus 100determines a panel type of the medical slide image. As described above,the panel types may be classified into a cell panel, a tissue panel, astructure panel, and the like. Example images of the cell panel type areshown in FIG. 10 , example images of the tissue panel are shown in FIG.11 , and example images of the structure panel are shown in FIG. 12 . Asshown in FIG. 10 to FIG. 12 , the cell panel is a patch type where acell-level annotation is performed, the tissue panel is a patch typewhere a tissue-level annotation is performed, and the structure panel isa patch type where an annotation, which is associated with a structureof a cell or tissue, is performed.

In some embodiments, the panel type may be determined by theadministrator.

In some embodiments, the panel type may be determined based on an outputvalue of the machine learning model. Referring to FIG. 13 , the machinelearning model may include a first machine learning model 75-1corresponding to a cell panel (that is, a model that learns a cell-levelannotation) and a second machine learning model 75-2 corresponding to atissue panel, and a third machine learning model 75-3 corresponding to astructure panel. In this case, the management apparatus 100 may extract(or sample) first, second, and third images 73-1, 73-2, and 73-3corresponding to the respective panels from a given medical slide image71, input each of the first, second, and third images 73-1, 73-2, and73-3 to each of the corresponding models 75-1, 75-2, and 75-3, and mayobtain output values 77-1, 77-2, and 77-3 as results. In addition, themanagement apparatus 100 may determine the panel type of the medicalslide image 71 by comparing the output values 77-1, 77-2, and 77-3 witha reference value. For example, when the first output value 77-1 is lessthan the reference value, the panel type of the medical slide image 71may be determined as the cell panel. This is because cell patchesextracted from the medical slide image 71 can effectively improve thelearning performance of the first machine learning model 75-1.

In some embodiments, the medical slide image may have a plurality ofpanel types. In this case, patches corresponding to each panel may beextracted from the medical slide image.

Referring back to FIG. 8 , in step S250, the management apparatus 100determines an annotation task. As described above, the annotation taskindicates an entity defined by the detailed job type.

In some embodiments, the annotation task may be determined by theadministrator.

In some embodiments, the annotation task may be automatically determinedbased on a combination of the determined dataset type and panel type.For example, when the annotation task corresponding to the combinationof the dataset type and the panel type is predefined, the correspondingannotation task may be automatically determined based on thecombination.

In step S270, the management apparatus 100 automatically extracts apatch on which the annotation is actually to be performed, from themedical slide image. In some embodiments, an area designated by theadministrator may be extracted as the patch. A specific method ofautomatically extracting the patch may vary depending on embodiments,and various embodiments related to the patch extraction will bedescribed with reference to FIG. 14 to FIG. 23 .

In some embodiments, although not shown in FIG. 8 , after step S270, themanagement apparatus 100 may generate an annotation job based on thedataset type, panel type, annotation task, and patch determined in stepsS210 to S270. As described above, the generated annotation job may beassigned to an account of an appropriate annotator.

The method of generating the annotation job according to someembodiments of the present disclosure has been described above withreference to FIG. 8 to FIG. 13 . Hereinafter, various embodiments of thepresent disclosure related to automatic patch extraction are describedwith reference to FIG. 14 to FIG. 23 .

FIG. 14 is an example flowchart showing an automatic patch extractionmethod according to a first embodiment of the present disclosure.However, the flowchart merely corresponds to an exemplary embodiment forachieving the object of the present disclosure, and some steps may beadded or omitted as necessary.

As shown in FIG. 14 , the automatic patch extraction method begins withstep S271 of sampling a plurality of candidate patches in a new medicalslide image. A specific process of sampling the plurality of candidatepatches may vary depending on embodiments.

In some embodiments, in a case that at least cell regions constituting aspecific tissue are sampled as candidate patches (that is, patches of acell panel type), a tissue area 83 may be extracted from a medical sliceimage 81 through image analysis, and a plurality of candidate patches 85may be sampled within the extracted area 83, as shown in FIG. 15 . Someexamples of sampling results are shown in FIG. 16 and FIG. 17 . Inmedical slide images 87 and 89 shown in FIG. 16 and FIG. 17 , each pointrefers to a sampling point and each of rectangular figures refers to asampling area (that is, a candidate patch area). As shown in FIG. 16 andFIG. 17 , the plurality of candidate patches may be sampled in such amanner that at least some of the candidate patches overlap with oneanother.

In some embodiments, the candidate patches may be generated by uniformlydividing an entire area of a medical slide image and then sampling eachof the divided areas. That is, the sampling may be performed in anequally dividing manner In this case, the size of each candidate patchmay be a predetermined fixed value, or a variable value determined basedon a size, resolution, panel type, or the like of the medical slideimage.

In some embodiments, the candidate patches may be generated by randomlydividing the entire area of the medical slide image and then samplingeach of the divided areas.

In some embodiments, the candidate patches may be configured such thatthe number of objects exceeds a reference value. For example, objectrecognition may be performed on the entire area of the medical slideimage, and, as a result, an area having a larger number of objects,which are calculated as a result of the object recognition, than thereference value may be sampled as a candidate patch. In such a case, thesizes of the candidate patches may be different.

In some embodiments, the candidate patches that are divided according toa policy determined based on metadata of the medical slide image may besampled. Here, the metadata may be a disease name, tissue or demographicinformation of a patient associated with the medical slide image, alocation of a medical institution, a quality (e.g. resolution) or formattype of the medical slide image, or the like. For example, when themedical slide image is an image about a tissue of a tumor patient,candidate patches may be sampled at cell level to be used as trainingdata of a machine learning model for mitotic cell detection. In anotherexample, in a case that lesion location in the tissue is critical whenthe prognosis of disease associated with the medical slide image isdiagnosed, candidate patches may be sampled at tissue level.

In some embodiments, when sampling candidate patches of the structurepanel type in the medical slide image, outlines are extracted from themedical slide image through image analysis, and sampling may beperformed so that the outlines that are connected to each other amongthe extracted outlines form one candidate patch.

As described above, a specific method of sampling a plurality ofcandidate patches in step S271 may vary depending on embodiments. Thedescription is continued referring back to FIG. 14 .

In step S273, an annotation target patch is selected based on an outputvalue of the machine learning model. The output value may be, forexample, a confidence score (or a confidence score of each class), and aspecific method of selecting a patch based on the confidence score mayvary depending on embodiments.

In some embodiments, the annotation target patch may be selected basedon the entropy value calculated by the confidence score for each class.Details of the present embodiment are shown in FIG. 18 and FIG. 19 .

As shown in FIG. 18 , annotation target patches 93 may be selected fromcandidate patches 92 that are sampled in a medical slide image 91,through uncertainty sampling based on entropy values. More specifically,as shown in FIG. 19 , entropy values 97-1 to 97-n may be calculatedbased on confidence scores for each class 96-1 to 96-n of candidatepatches 94-1 to 94-n that are output from a machine learning model 95.As described above, the entropy value has a larger value as theconfidence scores are more evenly distributed over classes. For example,in the case shown in FIG. 19 , entropy A (97-1) has the largest valueand entropy C (97-n) has the smallest value. In addition, the candidatepatches which have larger entropy value than a reference value may beautomatically selected as the annotation target. The patch having thelarger entropy value means that prediction results of the machinelearning model are uncertain, which means that the patch is moreeffective data for training.

In some embodiments, the annotation target patch may be selected basedon the confidence score itself. For example, among a plurality of thecandidate patches, a candidate patch having a confidence score less thana reference value may be selected as the annotation target patch.

FIG. 20 is an example flowchart showing an automatic patch extractionmethod according to a second embodiment of the present disclosure.However, the flowchart shown in FIG. 20 merely corresponds to anexemplary embodiment for achieving an object of the present disclosure,and some steps may be added or omitted as necessary. For convenience ofdescription, repeated descriptions of the above described embodimentsare omitted.

As shown in FIG. 20 , the second embodiment also begins with step S271of sampling a plurality of candidate patches. However, differently fromthe first embodiment, in the second embodiment, the annotation targetpatch is selected based on a misprediction probability of a machinelearning model (S275).

The misprediction probability of the machine learning model may becalculated based on a misprediction probability calculation model(hereinafter, referred as “calculation model”) that is constructedthrough machine learning. For convenience of understanding, a method ofconstruction the calculation model is described with reference to FIG.21 and FIG. 22 .

As shown in FIG. 21 , the calculation model may be constructed bylearning evaluation results (e.g. validation results and test results)of the machine learning model (S291 to S295). Specifically, when themachine learning model is evaluated with data for evaluation (S291) andthe evaluation result is tagged with label information about the datafor evaluation (S293), the calculation model may be constructed bylearning the tagged label information on the data for evaluation (S295).

Some examples of tagging label information to the data for evaluationare shown in FIG. 22 . FIG. 22 represents a confusion matrix. When themachine learning model is a model for performing a classification task,the evaluation result may correspond to a specific cell in the confusionmatrix. As shown in FIG. 22 , a first value (e.g., 0) is tagged as alabel 102 on an image 101 which has an evaluation result of FP (falsepositive) or FN (false negative), and a second value (e.g., 1) is taggedas a label 104 on an image 103 which has an evaluation result of TP(true positive) of TN (true negative). That is, when the prediction ofthe machine learning model is a correct answer, “1” may be tagged, and,otherwise, “0” may be tagged.

After learning the images 101 and 102 and the label information thereof,the calculation model outputs a high confidence score when an imagesimilar to the image correctly predicted by the machine learning modelis input and, otherwise, outputs a low confidence score. Therefore, thecalculation model can calculate a misprediction probability of themachine learning model for the input image.

Meanwhile, FIG. 22 merely shows some example for tagging the labelinformation. According to some embodiments of the present disclosure, aprediction error may be tagged as label information. Here, theprediction error represents a difference between a predicted value (thatis, confidence score) and an actual value (that is, correct answerinformation).

Further, according to some embodiments of the present disclosure, afirst value (e.g., 0) may be tagged when the prediction error of animage for evaluation is greater than or equal to a reference value, andotherwise a second value (e.g., 1) may be tagged.

Referring back to FIG. 20 , the description will be continued.

When the calculation model is constructed according to above-describedprocess, in step S275, the management apparatus 100 may calculate amisprediction probability for each of the plurality of candidatepatches. For example, as shown in FIG. 23 , the management apparatus 100inputs each of data samples 111-1 to 111-n into a calculation model 113to obtain the confidence score 115-1 to 115-n of the calculation model113, and calculates the misprediction probability based on the obtainedconfidence scores 115-1 to 115-n.

However, as shown in FIG. 23 , when the candidate patches 111-1 to 111-nare input, in a case that the calculation model 113 has been learned tooutput the confidence scores 115-1 to 115-n for each of correct classesand incorrect classes (that is, in a case that the calculation model hasbeen learned to tag label “1” when the prediction matches the correctanswer, and otherwise to tag label “0”), the confidence scores of theincorrect classes (shown as underlined) may be used as the mispredictionprobability.

When the misprediction probability of each of the candidate patches iscalculated, the management apparatus 100 may select as the annotationtarget a candidate patch, having a calculated misprediction probabilityequal to or greater than a reference value, from the plurality ofcandidate patches. A high misprediction probability means that theprediction results of the machine learning model are likely to beincorrect since the corresponding patches are important data forimproving the performance of the machine learning model. Thus, whenpatches are selected based on the misprediction probability,high-quality training datasets may be generated because the patcheseffective for learning are selected as the annotation targets.

The automatic patch extraction method according to various embodimentsof the present disclosure has been described above with reference toFIG. 14 to FIG. 23 . According to the above-described method, patchesindicating areas where annotations are to be performed can beautomatically extracted. Therefore, the workload of the administratorcan be minimized In addition, since only patches effective for learningamong a plurality of candidate patches are selected as annotationtargets based on misprediction probability or entropy values of themachine learning model, the amount of annotation job can be reduced, anda high-quality training dataset can be generated.

Hereinafter, an example computing device 200 that may implement anapparatus (e.g. management apparatus 100)/system according to variousembodiments of the present disclosure will be described with referenceto FIG. 24 .

FIG. 24 is an example hardware block diagram illustrating an examplecomputing device 200 that can implement an apparatus according tovarious embodiments of the present disclosure.

As shown in FIG. 24 , the computing device 200 may include one or moreprocessors 210, a bus 250, a communication interface 270, a memory 230to which a computer program executed by the processor 210 is loaded, anda storage 290 that stores a computer program 291. However, FIG. 24 showsmerely elements related to the embodiments of the present disclosure.Therefore, the person of ordinary skill in the art will understand thatgeneral elements other that those shown in FIG. 24 may be furtherincluded.

The processor 210 controls the overall operation of each component ofthe computing device 200. The processor 210 may configured to include atleast one of a central processing unit (CPU), a micro processor unit(MPU), a micro controller unit (MCU), a graphics processing unit (GPU),or any form of processor well known in the technical field of thepresent disclosure. The processor 210 may perform calculation of atleast one application or program for executing methods or operationsaccording to embodiments of the present disclosure.

The memory 230 stores various data, commands, and/or information. Toexecute methods or operations according to various embodiments of thepresent disclosure, the memory 230 may load one or more programs 291from the storage 290. The memory 230 may be implemented as a volatilememory such as a random access memory (RAM), but the technical scope ofthe present disclosure is not limited thereto.

The bus 250 provides a communication function between elements of thecomputing device 200. The bus 250 may be implemented as various forms ofbuses, such as an address bus, a data bus, and a control bus, and thelike.

The communication interface 270 supports wired or wireless Internetcommunication of the computing device 200. Further, the communicationinterface 270 may support various communication methods as well asInternet communication. To this end, the communication interface 270 mayinclude a communication module well known in the technical field of thepresent disclosure.

The storage 290 may non-temporarily store the one or more programs 291.The storage 290 may include a non-volatile memory, such as Read OnlyMemory (ROM), an eraseable Programmable ROM (EPROM), an electricallyerasable programmable ROM (EEPROM), and a flash memory, a hard disk, aremovable disk, or any form of computer-readable recording medium wellknown in the art to which the present disclosure pertains.

Computer program 291 may include one or more instructions which causethe processor 210 to perform methods or operations according to variousembodiments of the present disclosure by performing the one or moreinstructions. In other words, the processor 210 may execute methods oroperations according to various embodiments of the present disclosure byperforming the one or more instructions.

For example, the computer program 291 may include one or moreinstructions to perform an operation of obtaining information about anew medical slide image, determining a dataset type and a panel of themedical slide image, and the pathological slide image, the determineddataset type, assigning an annotation task and annotation job, which isdefined by a patch that is a part of the medical slide image, to anaccount of an annotator. In this case, the management apparatus 100according to some embodiments of the present disclosure may beimplemented through the computing device 200.

An example computing device that may implement apparatuses according tovarious embodiments of the present disclosure has been described abovewith reference to FIG. 24 .

The concepts of the disclosure described above with reference to FIG. 1to FIG. 24 may be embodied as computer-readable code on acomputer-readable medium. The computer-readable medium may be, forexample, a removable recording medium (a CD, a DVD, a Blu-ray disc, aUSB storage device, or a removable hard disc) or a fixed recordingmedium (a ROM, a RAM, or a computer-embedded hard disc). The computerprogram recorded on the computer-readable recording medium may betransmitted to another computing apparatus via a network such as theInternet and installed in another computing device, so that the computerprogram can be used in another computing device.

The technical concept of the present disclosure is not necessarilylimited to these embodiments, as all the elements configuring theembodiments of the present disclosure have been described as beingcombined or operated in combination. That is, within the scope of thepresent disclosure, all of the elements may be selectively operable incombination with one or more.

Although operations are shown in a specific order in the drawings, itshould not be understood that desired results can be obtained when theoperations must be performed in the specific order or sequential orderor when all of the operations must be performed. In certain situations,multitasking and parallel processing may be advantageous. According tothe above-described embodiment, it should not be understood that theseparation of various configurations is necessarily required, and itshould be understood that the described program components and systemsmay generally be integrated together into a single software product orbe packaged into multiple software products.

While the present disclosure have been particularly illustrated anddescribed with reference to embodiments thereof, it will be understoodby a person of ordinary skill in the art that various changes in formand detail may be made therein without departing from the spirit andscope of the present disclosure as defined by the following claims. Theembodiments should be considered in a descriptive sense only and not forpurposes of limitation.

1. A terminal comprising: a memory including one or more instructions;and a processor configured to, by executing the one or moreinstructions, display an annotation tool including a plurality ofselectable tools respectively corresponding to a plurality ofannotations and an area in which a partial area of the medical slideimage is displayed, receive an input of an annotator through theannotation tool, and perform an annotation on the partial area based onthe input of the annotator, wherein the annotation is used for a machinelearning model to perform a target task, and wherein the target taskincludes classifying at least one of a plurality of classes including aclass related to a cell and a class related to a tissue. 2-3. (canceled)4. The terminal of claim 1, wherein the annotation tool further includesinformation related to an annotation job.
 5. The terminal of claim 4,wherein the processor is configured to change the plurality ofselectable tools based on information of the annotation job.
 6. Theterminal of claim 1, wherein the area further includes an indicator forcontrolling enlargement or reduction of the partial area.
 7. Theterminal of claim 1, further comprising a communication interface,wherein the processor is further configured to receive, via thecommunication interface, the annotation job for the medical slide imagefrom a management apparatus, and provide a result of the annotation tothe management apparatus, and wherein the result of the annotationincludes label information tagged in the partial area.
 8. A managementapparatus comprising: a communication interface; a memory including oneor more instructions; and a processor configured to, by executing theone or more instructions, transmit, via the communication interface, anannotation job for a medical slide image to a terminal of an annotator,receive, via the communication interface, a result of the annotation jobperformed through an annotation tool from the terminal, wherein theannotation tool includes a plurality of selectable tools respectivelycorresponding to a plurality of annotations and an area in which apartial area of the medical slide image is displayed, the partial areabeing an area on which the annotation is performed in the medical sliceimage, and use the result of the annotation job for a first machinelearning model to perform a target task, wherein the target taskincludes classifying at least one of a plurality of classes including aclass related to a cell and a class related to a tissue.
 9. Themanagement apparatus of claim 8, wherein the processor is furtherconfigured to perform validation on the result of the annotation job,and record a result of the validation as an evaluation result of theannotator.
 10. The management apparatus of claim 9, wherein theprocessor is configured to: input the medical slide image to a secondmachine learning model; and perform the validation on the result of theannotation job based on comparing the result of the annotation job witha result output from the second machine learning model.
 11. Themanagement apparatus of claim 10, wherein the processor is configuredto: select another annotator in response to the result of the validationindicating that the annotation is performed again; and transmits theannotation job to a terminal of the another annotator. 12-16. (canceled)17. The management apparatus of claim 8, wherein the result of theannotation job includes label information tagged in the partial area.18. A method performed by a terminal, the method comprising: displayingan annotation tool including a plurality of selectable toolsrespectively corresponding to a plurality of annotations and an area inwhich a partial area of the medical slide image is displayed; receivingan input of an annotator through the annotation tool; and performing anannotation on the partial area based on the input of the annotator,wherein the annotation is used for a machine learning model to perform atarget task, and wherein the target task includes classifying at leastone of a plurality of classes including a class related to a cell and aclass related to a tissue.
 19. The terminal of claim 1, whereinperforming the annotation based on the input of the annotator includes:selecting, as a selected tool, a selectable tool from among theplurality of selectable tools based on a first input of the annotator,and performing the annotation corresponding to the selected tool amongthe plurality of different annotations based on a second input of theannotator.
 20. The terminal of claim 1, wherein the plurality ofselectable tools are differentiated by different colors.
 21. Theterminal of claim 1, wherein the partial area is highlighted by arectangle shape in the medical slide image.
 22. The management apparatusof claim 8, wherein the result of the annotation job includes anannotation corresponding to a selected tool among the plurality ofdifferent annotations, the selected tool being selected from among theplurality of selectable tools.
 23. The management apparatus of claim 8,wherein the plurality of selectable tools are differentiated bydifferent colors.
 24. The management apparatus of claim 8, wherein thepartial area is highlighted by a rectangle shape in the medical sliceimage.
 25. The method of claim 18, wherein performing the annotationbased on the input of the annotator includes: selecting, as a selectedtool, a selectable tool from among the plurality of selectable toolsbased on a first input of the annotator, and performing the annotationcorresponding to the selected tool among the plurality of differentannotations based on a second input of the annotator.
 26. The method ofclaim 18, wherein the plurality of selectable tools are differentiatedby different colors.
 27. The method of claim 18, wherein the partialarea is highlighted by a rectangle shape in the medical slice image.